Using Ground Truths to Improve Wisdom of the Crowd Estimates

In this paper we explore a cognitive modeling approach to aggregating individuals’ estimates of unknown quantities without natural bounds. We carried out two experiments that elicited individuals’ estimates of the population of US metropolitan areas, and domestic box office returns for movies. We found that the means of individuals’ responses correlate well with the true sizes, but participants systematically underestimated these values. We formulated a cognitive model that uses the true values of known items to correct for individuals’ biases, and demonstrated that this model can drastically improve predictive accuracy. Because our model quantitatively infers individual’s biases on the estimation tasks we were able to examine the distribution of individual biases, and found that there were substantial between-individual differences in the magnitude of the responses. This work demonstrates how individuals’ biases, whether overor underestimation, can be corrected using a cognitive model together with known ground truths.